{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FARM Building Blocks\n",
    "\n",
    "Welcome to the FARM building blocks tutorial! There are many different ways to make use of this repository, but in this notebook, we will be going through the most import building blocks that will help you harvest the rewards of a successfully trained NLP model.\n",
    "\n",
    "Happy FARMing!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Text Classification\n",
    "\n",
    "GermEval 2018 (GermEval2018) (https://projects.fzai.h-da.de/iggsa/) is an open data set containing texts that need to be classified by whether they are offensive or not. There are a set of coarse and fine labels, but here we will only be looking at the coarse set which labels each example as either OFFENSE or OTHER. To tackle this task, we are going to build a classifier that is composed of Google's BERT language model and a feed forward neural network prediction head."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's start by adjust the working directory so that it is the root of the repository\n",
    "# This should be run just once.\n",
    "\n",
    "import os\n",
    "os.chdir('../')\n",
    "print(\"Current working directory is {}\".format(os.getcwd()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here are the imports we need\n",
    "\n",
    "import torch\n",
    "from farm.modeling.tokenization import Tokenizer\n",
    "from farm.data_handler.processor import TextClassificationProcessor\n",
    "from farm.data_handler.data_silo import DataSilo\n",
    "from farm.modeling.language_model import LanguageModel\n",
    "from farm.modeling.prediction_head import TextClassificationHead\n",
    "from farm.modeling.adaptive_model import AdaptiveModel\n",
    "from farm.modeling.optimization import initialize_optimizer\n",
    "from farm.train import Trainer\n",
    "from farm.utils import MLFlowLogger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Farm allows simple logging of many parameters & metrics. Let's use MLflow framework to track our experiment ...\n",
    "ml_logger = MLFlowLogger(tracking_uri=\"https://public-mlflow.deepset.ai/\")\n",
    "ml_logger.init_experiment(experiment_name=\"Public_FARM\", run_name=\"Tutorial1_Colab\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to fetch the right device to drive the growth of our model\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "device = torch.device(\"cpu\")\n",
    "print(\"Devices available: {}\".format(device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Handling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we initialize a tokenizer that will be used for preprocessing text\n",
    "# This is the BERT Tokenizer which uses the byte pair encoding method.\n",
    "# It is currently loaded with a German model\n",
    "\n",
    "tokenizer = Tokenizer.load(\n",
    "    pretrained_model_name_or_path=\"bert-base-german-cased\",\n",
    "    do_lower_case=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In order to prepare the data for the model, we need a set of\n",
    "# functions to transform data files into PyTorch Datasets.\n",
    "# We group these together in Processor objects.\n",
    "# We will need a new Processor object for each new source of data.\n",
    "# The abstract class can be found in farm.data_handling.processor.Processor\n",
    "\n",
    "processor = TextClassificationProcessor(tokenizer=tokenizer,\n",
    "                                        max_seq_len=128,\n",
    "                                        data_dir=\"data/germeval18\",\n",
    "                                        label_list = [\"OTHER\", \"OFFENSE\"],\n",
    "                                        metric = \"f1_macro\",\n",
    "                                        label_column_name = \"coarse_label\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need a DataSilo in order to keep our train, dev and test sets separate.\n",
    "# The DataSilo will call the functions in the Processor to generate these sets.\n",
    "# From the DataSilo, we can fetch a PyTorch DataLoader object which will\n",
    "# be passed on to the model.\n",
    "# Here is a good place to define a batch size for the model\n",
    "\n",
    "BATCH_SIZE = 32\n",
    "\n",
    "data_silo = DataSilo(\n",
    "    processor=processor,\n",
    "    batch_size=BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In FARM, we make a strong distinction between the language model and prediction head so that you can mix and match different building blocks for your needs.\n",
    "\n",
    "For example, in the transfer learning paradigm, you might have the one language model that you will be using for both document classification and NER. Or you perhaps you have a pretrained language model which you would like to adapt to your domain, then use for a downstream task such as question answering. \n",
    "\n",
    "All this is possible within FARM and requires only the replacement of a few modular components, as we shall see below.\n",
    "\n",
    "Let's first have a look at how we might set up a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The language model is the foundation on which modern NLP systems are built.\n",
    "# They encapsulate a general understanding of sentence semantics\n",
    "# and are not specific to any one task.\n",
    "\n",
    "# Here we are using Google's BERT model as implemented by HuggingFace. \n",
    "# The model being loaded is a German model that we trained. \n",
    "# You can also change the MODEL_NAME_OR_PATH to point to a BERT model that you\n",
    "# have saved or download one connected to the HuggingFace repository.\n",
    "# See farm.modeling.language_model.PRETRAINED_MODEL_ARCHIVE_MAP for a list of\n",
    "# available models\n",
    "\n",
    "MODEL_NAME_OR_PATH = \"bert-base-german-cased\"\n",
    "\n",
    "language_model = LanguageModel.load(MODEL_NAME_OR_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A prediction head is a model that processes the output of the language model\n",
    "# for a specific task.\n",
    "# Prediction heads will look different depending on whether you're doing text classification\n",
    "# Named Entity Recognition (NER), question answering or some other task.\n",
    "# They should generate logits over the available prediction classes and contain methods\n",
    "# to convert these logits to losses or predictions \n",
    "\n",
    "# Here we use TextClassificationHead which receives a single fixed length sentence vector\n",
    "# and processes it using a feed forward neural network. layer_dims is a list of dimensions:\n",
    "# [input_dims, hidden_1_dims, hidden_2_dims ..., output_dims]\n",
    "\n",
    "# Here by default we have a single layer network.\n",
    "# It takes in a vector of length 768 (the default size of BERT's output).\n",
    "# It outputs a vector of length 2 (the number of classes in the GermEval18 (coarse) dataset)\n",
    "\n",
    "LAYER_DIMS = [768, 2]\n",
    "\n",
    "prediction_head = TextClassificationHead(layer_dims=LAYER_DIMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The language model and prediction head are coupled together in the Adaptive Model.\n",
    "# This class takes care of model saving and loading and also coordinates\n",
    "# cases where there is more than one prediction head.\n",
    "\n",
    "# EMBEDS_DROPOUT_PROB is the probability that an element of the output vector from the\n",
    "# language model will be set to zero.\n",
    "EMBEDS_DROPOUT_PROB = 0.1\n",
    "\n",
    "model = AdaptiveModel(\n",
    "    language_model=language_model,\n",
    "    prediction_heads=[prediction_head],\n",
    "    embeds_dropout_prob=EMBEDS_DROPOUT_PROB,\n",
    "    lm_output_types=[\"per_sequence\"],\n",
    "    device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we initialize a Bert Adam optimizer that has a linear warmup and warmdown\n",
    "# Here you can set learning rate, the warmup proportion and number of epochs to train for\n",
    "\n",
    "LEARNING_RATE = 2e-5\n",
    "N_EPOCHS = 1\n",
    "\n",
    "model, optimizer, lr_schedule = initialize_optimizer(\n",
    "    model=model,\n",
    "    device=device,\n",
    "    learning_rate=LEARNING_RATE,\n",
    "    n_batches=len(data_silo.loaders[\"train\"]),\n",
    "    n_epochs=N_EPOCHS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training loop handled by this\n",
    "# It will also trigger evaluation during training using the dev data\n",
    "# and after training using the test data.\n",
    "\n",
    "# Set N_GPU to a positive value if CUDA is available\n",
    "N_GPU = 0\n",
    "\n",
    "trainer = Trainer(\n",
    "    optimizer=optimizer,\n",
    "    data_silo=data_silo,\n",
    "    epochs=N_EPOCHS,\n",
    "    n_gpu=N_GPU,\n",
    "    lr_schedule=lr_schedule,\n",
    "    device=device,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = trainer.train(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test your model on a sample (Inference)\n",
    "from farm.infer import Inferencer\n",
    "from pprint import PrettyPrinter\n",
    "\n",
    "infer_model = Inferencer(processor=processor, model=model, gpu=True)\n",
    "\n",
    "basic_texts = [\n",
    "    {\"text\": \"Martin ist ein Idiot\"},\n",
    "    {\"text\": \"Martin Müller spielt Handball in Berlin\"},\n",
    "]\n",
    "result = infer_model.inference_from_dicts(dicts=basic_texts)\n",
    "PrettyPrinter().pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Switch to NER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a transfer learning paradigm, there is a core computation that is shared amongst all tasks. FARM's modular structure means that you can easily swap out different building blocks to make the same language model work for many different tasks.\n",
    "\n",
    "We can adapt the above text classification model to NER by simply switching out the processor and prediction head."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the new building blocks\n",
    "\n",
    "from farm.data_handler.processor import NERProcessor\n",
    "from farm.modeling.prediction_head import TokenClassificationHead\n",
    "ml_logger.init_experiment(experiment_name=\"Public_FARM\", run_name=\"Tutorial1_Colab_NER\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This processor will preprocess the data for the CoNLL03 NER task\n",
    "ner_labels = [\"[PAD]\", \"X\", \"O\", \"B-MISC\", \"I-MISC\", \"B-PER\", \"I-PER\", \"B-ORG\", \"I-ORG\", \"B-LOC\", \"I-LOC\", \"B-OTH\", \"I-OTH\"]\n",
    "\n",
    "ner_processor = NERProcessor(tokenizer=tokenizer, \n",
    "                             max_seq_len=128, \n",
    "                             data_dir=\"../data/conll03-de\",\n",
    "                             label_list=ner_labels,\n",
    "                             metric=\"seq_f1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This prediction head is also a feed forward neural network but expects one\n",
    "# vector per token in the input sequence and will generate a set of logits\n",
    "# for each input\n",
    "\n",
    "LAYER_DIMS = [768, 13]\n",
    "\n",
    "ner_prediction_head = TokenClassificationHead(layer_dims=LAYER_DIMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can integrate these new pieces with the rest using this code\n",
    "# It is pretty much the same structure as what we had above for text classification\n",
    "\n",
    "BATCH_SIZE = 32\n",
    "EMBEDS_DROPOUT_PROB = 0.1\n",
    "LEARNING_RATE = 2e-5\n",
    "N_EPOCHS = 1\n",
    "N_GPU = 1\n",
    "\n",
    "data_silo = DataSilo(\n",
    "    processor=ner_processor,\n",
    "    batch_size=BATCH_SIZE)\n",
    "\n",
    "model = AdaptiveModel(\n",
    "    language_model=language_model,\n",
    "    prediction_heads=[ner_prediction_head],\n",
    "    embeds_dropout_prob=EMBEDS_DROPOUT_PROB,\n",
    "    lm_output_types=[\"per_token\"],\n",
    "    device=device)\n",
    "\n",
    "model, optimizer, lr_schedule = initialize_optimizer(\n",
    "    model=model,\n",
    "    learning_rate=LEARNING_RATE,\n",
    "    n_batches=len(data_silo.loaders[\"train\"]),\n",
    "    n_epochs=N_EPOCHS)\n",
    "\n",
    "trainer = Trainer(\n",
    "    optimizer=optimizer,\n",
    "    data_silo=data_silo,\n",
    "    epochs=N_EPOCHS,\n",
    "    n_gpu=N_GPU,\n",
    "    lr_schedule=lr_schedule,\n",
    "    device=device,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = trainer.train(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}